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Abstract 

Background: Using a different mode of contact on the final follow-up to survey non-respondents is an identified 
strategy to increase response rates. This study was designed to determine if a reminder phone call or a phone 
interview as a final mode of contact to a mailed survey works better to increase response rates and which strategy 
is more cost effective. 

Methods: A randomized study was embedded within a survey study of individuals treated with ulcerative colitis 
conducted in March 2009 in Olmsted County, Minnesota. After two mail contacts, non-respondents were randomly 
assigned to either a reminder telephone call or a telephone interview. Average cost per completed interview and 
response rates were compared between the two experimental conditions. 

Results: The response rate in the reminder group and the interview did not differ where we considered both a 
completed survey and a signed form a complete (24% vs. 29%, p = 0.08). However, if such a signed form was not 
required, there was a substantial advantage to completing the interview over the phone (24% vs. 43%, p < 0.0001). 
The reminder group on average cost $27.00 per completed survey, while the interview group on average cost 
$53.00 per completed survey when a signed form was required and $36.00 per complete when a signed form was 
not required. 

Conclusions: The additional cost of completing an interview is worth it when an additional signed form is not 
required of the respondent. However, when such a signed form is required, offering an interview instead of a 
reminder phone call as a follow up to non-respondents does not increase response rates enough to outweigh the 
additional costs. 

Keywords: Survey methods, Telephone follow-up, Health surveys, Survey Nonresponse follow-up 



Background 

Although response rates vary by study and mode of 
administration, there is evidence that response rates are 
declining across all modes of survey administration 
[1-3]. Mixed mode surveys have been shown to increase 
response rates by appealing to respondents who did not 
respond to the first mode they were offered [4]. Thus 
switching modes can offer potential respondents an 
additional opportunity for survey completion and allow 
respondents to complete the survey utilizing a mode 
with which they may feel more comfortable. 
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Dillman and colleagues [5] suggest making the final 
contact to a potential respondent distinct from previous 
failed attempts as a way to increase response rates based 
on the premise that people prefer different modes of 
data collection. While historically in mail surveys this 
took the form of final contact by certified mail or more 
recently delivery service, they recommend using a tele- 
phone contact when telephone numbers are available 
[5]. In a recent meta-analysis, telephone reminders were 
found to have no overall impact on response rate com- 
pared to no reminder [6]. However, the results are 
somewhat inconclusive as they are only based on three 
studies and the findings are in opposing directions [6]. 
Although generally more expensive than additional mail 
contacts on a per unit basis, telephone reminders can 
result in reduced costs when considering cost per 
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completed survey. One study found a telephone remin- 
der was associated with a significant increase in 
response rates and with a marginal cost savings of 15% 
lower costs per completed survey compared to when no 
reminder phone call was used [7]. 

Conventional wisdom in the survey research field dic- 
tates that once a potential respondent is on the phone 
for a reminder, completing an interview over the phone 
is an attractive option as not doing so may present a 
missed opportunity. In fact, Dillman and colleagues 
further suggest completing an interview over the tele- 
phone as a mechanism to increase response rates, albeit 
with increased implementation and completion costs 
due to the use of interviewers to administer the survey 
versus merely reminding prospective respondents to fill 
out the mailed form [5]. In this study, using a rando- 
mized design, we compare the relative effectiveness and 
cost per complete survey of phone reminders compared 
to phone interviews at the third contact of a mailed 
survey. 

Methods 

The present methodological study was embedded within 
a larger study conducted in March 2009 by Holubar and 
colleagues assessing the influence of referral bias on 
patient reported outcomes among patients treated for 
ulcerative colitis [8]. The survey was conducted in 
Olmsted County, Minnesota based on a frame estab- 
lished using the resources of the Rochester Epidemiol- 
ogy Project (REP), the medical record system for health 
care providers to residents of Olmsted County [9]. The 
REP was used to identify study patients who were either 
a resident of Olmsted County or referral-center patients 
who underwent a surgical procedure (ileal pouch-anal 
anastomosis or IPAA) in the past 20 years. Respondents 
were selected from the REP (N = 1825) and after two 
mailings, 873 non-responders were randomized to one 
of two conditions: a telephone reminder (N = 438) or a 
telephone interview (N = 435). This study was approved 
by the Mayo Clinic and Olmsted Clinic Institutional 
Review Boards. 

Analytical strategy 

In the larger study, only surveys with both a signed 
Health Insurance Portability and Accountability Act 
(HIPAA) authorization form (HAF) and a completed 
interview or survey were considered complete. For the 
purposes of this analysis and to make the results gener- 
alizable to institutions that do not require a signed 
form, we consider two definitions of what it means to 
be a respondent: 1) generalizable respondent: completed 
survey or interview regardless of HAF status, and 2) 
study respondent: completed survey or interview and 
signed form returned. While a survey could be 



completed only in the paper mode within the reminder 
group, it could be completed either as a telephone inter- 
view or by paper within the interview group if a respon- 
dent chose to not complete the phone interview even if 
they were randomized to that arm. Response rates and 
costs were compared between reminder and interview 
groups under each definition of respondent (generaliz- 
able and study respondent). Marginal cost per com- 
pleted survey was calculated by multiplying the cost per 
hour of telephone interviewers (who also completed the 
reminder phone calls) by the total number of hours 
worked by the interviewers in each condition. 

The response rates for both the generalizable and 
study respondent definitions were compared between 
the two conditions overall and within age and gender 
subpopulations using Chi-square tests of independence. 
All analyses were conducted using SAS 9.1 statistical 
software. Unadjusted p-values are reported. Within each 
set of analyses, 14 total statistical tests were performed. 
P-values less than 0.0036 (= 0.05/14) were considered 
statistically significant using a Bonferroni adjustment for 
multiple testing. Raw p-values less than 0.05 were also 
noted as indicating potentially important differences. All 
variables were treated as categorical including age, gen- 
der, and return of signed form. 

Results 

Overall 9% of individuals refused the survey. An addi- 
tional 30% could not be contacted or were lost to follow- 
up. These outcomes did not differ across the interview 
and reminder groups. The survey response rate (Table 1) 
in the reminder group and the interview group were not 
significantly different using our definition of a study 
respondent where we received both a completed survey 
and a signed form (24% vs. 29%, p = 0.08). The reminder 
group on average cost $27.00 per completed survey, 
while the interview group on average cost $53.00 per 
completed survey. In contrast, there was a significant dif- 
ference in response rate when using the generalizable 
definition of a respondent that included a survey 
response regardless of receipt or non-receipt of a signed 
form (24% vs. 43%, p < 0.0001). The cost per complete in 
the reminder group was still $27.00 while the cost per 
complete in the interview group was reduced to $36.00. 

Using either definition of respondent, response rates 
for completed surveys were not significantly different 
within reminder and interview group respectively for 
either age or gender (Table 1). There were no between 
group differences using the study definition of respon- 
der. There were, however higher response rates for 
some subpopulations using the generalizable definition 
of respondent. Specifically, the response rate was higher 
in the interview group for both males and females as 
well as those individuals 35-44 and 55-64. (Table 1). 



Ziegenfuss et al. BMC Medical Research Methodology 2012, 12:32 
http://www.biomedcentral.com/1471-2288/12/32 



Page 3 of 4 



Table 1 Response rate by respondent population and demographic subpopulations 





Interview Group (n = 


435) Reminder Group (n 


= 438) 






n (%) response 


p* n (%) response 


P* 


p** 


STUDY RESPONDENT: Completed Survey and HIPAA Authorization Form 


Total 


1 28 (29.4%) 


1 06 (24.2%) 




0.08 


Mail Survey 


23 (5.3%) 


1 06 (24.2%) 




< 0.0001 


Phone Interview 


105 (24.1%) 


NA 




(only applies to interview group) 


Gender 


Male 


65 (26.2%) 


0.09 65 (23.5%) 


0.64 


0.47 


Female 


63 (33.7%) 


41 (25.5%) 




0.10 


Age Group 


18-24 


6 (31.6%) 


0.03 4 (36.4%) 


0.01 


0.79 


25-34 


18 (26.1%) 


22 (29.0%) 




0.70 


35-44 


23 (20.4%) 


15 (11.8%) 




0.07 


45-54 


37 (28.9%) 


34 (28.3%) 




0.92 


55-64 


32 (42.1%) 


22 (31.4%) 




0.18 


65+ 


1 2 (40.0%) 


9 (26.5%) 




0.25 


GENERALIZABLE RESPONDENT: Completed Survey Regardless of HIPAA Authorization Form 


Total 


1 87 (43.0%) 


1 06 (24.2%) 




< 0.0001 


Mail Survey 


24 (5.5%) 


1 06 (24.2%) 




0.50 


Phone Interview 


1 63 (37.5%) 


NA 




(only applies to interview group) 


Gender 


Male 


97 (39.1%) 


0.06 65 (23.5%) 


0.64 


< 0.001 


Female 


90 (48.1%) 


41 (25.5%) 




< 0.001 


Age Group 


18-24 


9 (47.4%) 


0.08 4 (36.4%) 


0.01 


0.56 


25-34 


28 (40.6%) 


22 (29.0%) 




0.14 


35-44 


39 (34.5%) 


15 (11.8%) 




< 0.001 


45-54 


53 (41 .4%) 


34 (28.3%) 




0.03 


55-64 


42 (55.3%) 


22 (31.4%) 




0.004 


65+ 


1 6 (53.3%) 


9 (26.5%) 




0.03 



* Within group 



Between groups 



Discussion 

The response rates were five percentage points higher 
for the interview group than the reminder group using 
the conservative definition of a respondent (i.e., received 
both a completed survey and a signed form; however 
this difference did not reach statistical significance. 
Thus, conclusions regarding whether the response rate 
was higher due to chance or due to the option of com- 
pleting the survey in an alternate mode (i.e. telephone) 
remains uncertain. What we did observe was a two-fold 
difference in the cost per completion in the interview 
versus reminder condition. Specifically, the interview 
group cost on average $53.00 per completed interview 
compared to an average of $27.00 per completed inter- 
view in the reminder group. Applying this difference to 
the entire sample, had we deployed an interviewing 



strategy in comparison to a reminder phone call as a 
matter of course, our total study cost would have been 
$7,844 (137%) more expensive. This could represent a 
financial challenge to investigators and studies con- 
strained by limited resources. 

In contrast, had we not been constrained to using only 
those surveys for which we also received a signed form 
we would have obtained a 19 percentage point increase 
in response rate in the interview group compared to the 
reminder group. This translated into only an additional 
$9.00 per complete in the interview group (for a total of 
$36.00 per complete). Thus, the decision to complete a 
telephone interview instead of just a reminder phone 
call hinges on whether or not prospective respondents 
are required to mail in a signed consent or authorization 
form along with the completed survey (the former 
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distinguishes our study definition of a respondent from 
the generalizable definition). 

We saw that in the case where a signed form was 
required, reminder telephone calls, while not negatively 
impacting response rates, were a more cost-effective 
way to achieve survey completion than offering a tele- 
phone interview. Moreover, reminder phone calls do not 
change the mode (i.e. self-administered via mail used in 
the first two contacts) of survey completion while inter- 
views do. Therefore potential response pattern differ- 
ences due to mode are nonexistent when using 
reminder calls, an additional benefit to reminders versus 
interviews. As there is evidence that response patterns 
can differ when an interviewer is present compared to a 
self- administered survey [4], implementing mixed mode 
data collection through a final telephone interview 
increases the risks of different response patterns across 
contact waves. 

Conclusions 

Although the strength and generalizability of our find- 
ings may be limited by the very specialized nature of 
our sample (patients who underwent ileal pouch-anal 
anastomosis), our study represents one of the first for- 
mal investigations assessing the relative merits of inter- 
viewing versus reminder call in bringing about enhanced 
participation. We conclude that offering a telephone 
interview instead of a reminder phone call as a follow- 
up to survey non-response did not increase response 
rates enough to outweigh the additional costs when a 
signed form is required. However, in instances where 
such a mail form is not required, it behooves research- 
ers to complete an interview with a respondent when 
they have them on the phone. 



References 

1. Hox JJ, de Leeuw ED: A comparison of nonresponse in mail, telephone, 
and face-to-face surveys: Applying multilevel modeling to metaanalysis. 

Qual Quant 1994, 28(4):329-344. 

2. Steeh C, Kurgis N, Cannon B, et al: Are they really as bad as they seem? 
Nonresponse rates at the end of the twentieth century. J Off Stat 2001, 
17(2):227-247. 

3. Hartge P: Raising response rates: Getting to yes. Epidemiology 1999, 
1 0(2):1 05-1 07. 

4. de Leeuw ED: To mix or not to mix data collection modes in surveys. J 

Off Stat 2005, 21(2):233-255. 

5. Dillman DA, Smyth JD, Christian LM: Internet, mail, and mixed-mode surveys: 
The tailored design method New Jersey: Wiley; 2009. 

6. Edwards PJ, Roberts I, Clarke MJ, et al: Methods to increase response to 
postal and electronic questionnaires. Cochrane Database Syst Rev 2009, 3. 

7. Silva SM, Smith WT, Bammer G: Telephone reminders are a cost effective 
way to improve responses in postal health surveys. J Epidemiol 
Community Health 2002, 56:1 15-1 18. 

8. Holubar SD, Cima RR, Pemberton JH, et al: Referral Bias: Is there a 
difference incontinence and quality of life after ileal pouch-anal 
anastomosis for ulcerative colitis between referral center and 
population-based cohorts in Olmsted County, MN. American Society of 
Colon and Rectal Surgeons (Annual Meeting Poster Abstracts) 201 0, 
53(4):94-95. 

9. Melton U: History of the Rochester Epidemiology Project. Mayo Clin Proc 
1996, 71(3):266-274. 

Pre-publication history 

The pre-publication history for this paper can be accessed here: 
http://www.biomedcentral.com/1471-2288/12/32/prepub 



doi:1 0.1 1 86/1 471-2288-1 2-32 

Cite this article as: Ziegenfuss et ah Telephone follow-up to a mail 
survey: when to offer an interview compared to a reminder call. BMC 

Medical Research Methodology 2012 12:32. 



Abbreviations 

REP: Rochester Epidemiology Project; HIPAA: Health Insurance Portability and 
Accountability Act; HAF: HIPAA authorization form. 

Author details 

'Division of Health Care Policy & Research, Mayo Clinic, 200 First Street SW, 
Rochester, MN, 55904, USA. Children's Hospital Boston, 300 Longwood Ave, 
Boston, MA, 02115, USA. department of General Surgery, Dartmouth- 
Hitchcock Medical Center, One Medical Center Drive, Lebanon, NH, 03756, 
USA. 

Authors' contributions 

AH conceptualized study design with assistance from TJB and SDH. TJB and 
AH oversaw study implementation. SDH led the parent study in which this 
study was embedded. JYZ directed analysis and interpretation of findings. 
KRB, AH and TJB participated in interpretation of findings. JYZ wrote the 
manuscript with assistance from KRB and TJB. All authors read and approved 
the final manuscript. 

Competing interests 

The authors declare that they have no competing interests. 

Received: 10 August 2011 Accepted: 20 March 2012 
Published: 20 March 2012 



Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 



Submit your manuscript at 
www.biomedcentral.com/submit 



BioMed Central 



v. 



